Collaborative airborne object tracking systems and methods

ABSTRACT

Systems and methods related to identifying locations and/or ranges to objects using airborne imaging devices are disclosed. An object tracking system may include a plurality of aerial vehicles having associated imaging devices and a control station. Information related to positions and orientations of aerial vehicles and associated imaging devices may be received. In addition, imaging data may be received and processed to identify optical rays associated with objects within the imaging data. Further, a three-dimensional mapping of the identified optical rays may be generated, and locations or ranges of the objects relative to the aerial vehicles may be determined based on any intersections of optical rays within the three-dimensional mapping.

BACKGROUND

Generally, aerial vehicles may include specialized hardware or sensorsto track objects in proximity and to determine relative ranges to suchobjects. For example, the specialized hardware or sensors may includeradar sensors, LIDAR sensors, laser rangefinders, or other similarspecialized hardware. However, such specialized hardware may addcomplexity, weight, and cost to aerial vehicles. Accordingly, there is aneed for systems and methods to track objects without the additionalcomplexity, weight, or cost of specialized hardware or sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

The detailed description is described with reference to the accompanyingfigures. In the figures, the left-most digit(s) of a reference numberidentifies the figure in which the reference number first appears. Theuse of the same reference numbers in different figures indicates similaror identical components or features.

FIG. 1 is a schematic diagram of an example object tracking system,according to an implementation.

FIG. 2 is a schematic diagram of an example aerial vehicle having animaging device, according to an implementation.

FIG. 3 is a schematic diagram of another example aerial vehicle havingan imaging device, according to an implementation.

FIG. 4 is a schematic diagram of an example field of view of an imagingdevice, according to an implementation.

FIGS. 5A and 5B are schematic diagrams of example aerial vehicles aspart of an object tracking system, according to an implementation.

FIG. 6 is a flow diagram illustrating an example object directional raydetermination process, according to an implementation.

FIG. 7 is a flow diagram illustrating an example object location andrange determination process, according to an implementation.

FIG. 8 is a block diagram illustrating various components of an aerialvehicle control system, according to an implementation.

FIG. 9 is a block diagram illustrating various components of an objecttracking control system, according to an implementation.

While implementations are described herein by way of example, thoseskilled in the art will recognize that the implementations are notlimited to the examples or drawings described. It should be understoodthat the drawings and detailed description thereto are not intended tolimit implementations to the particular form disclosed but, on thecontrary, the intention is to cover all modifications, equivalents andalternatives falling within the spirit and scope as defined by theappended claims. The headings used herein are for organizationalpurposes only and are not meant to be used to limit the scope of thedescription or the claims. As used throughout this application, the word“may” is used in a permissive sense (i.e., meaning having the potentialto), rather than the mandatory sense (i.e., meaning must). Similarly,the words “include,” “including,” and “includes” mean including, but notlimited to.

DETAILED DESCRIPTION

Systems and methods to track objects using imaging devices associatedwith aerial vehicles are described. The objects may be captured withinimaging data using imaging devices of a plurality of aerial vehicles.Location and/or pose information associated with the plurality of aerialvehicles, optical axis information and/or imaging characteristicsassociated with the imaging devices, and/or image processing techniquesmay be used in order to determine locations and/or ranges of the objectswithin the imaging data relative to at least some of the plurality ofaerial vehicles.

For example, the object tracking systems described herein may include aplurality of aerial vehicles and a control station. The plurality ofaerial vehicles may be any type of aerial vehicle, e.g., unmanned orautomated aerial vehicles such as quadcopters, hexacopters, octocopters,or other configurations. In addition, each of the plurality of aerialvehicles may include at least one imaging device, e.g., an analogcamera, a digital camera, a high-resolution camera, or other types ofimaging devices. Further, each of the plurality of aerial vehicles mayinclude an imaging device controller to modify a pose or orientation ofthe imaging device, and may also include an image processing module toperform image processing on the imaging data.

The plurality of aerial vehicles may determine and/or provide locationinformation using global positioning sensors and/or other locationsensors. In addition, the plurality of aerial vehicles may determineand/or provide pose information using inertial measurement units,accelerometers, and/or gyroscopes. Further, the plurality of aerialvehicles may determine and/or provide optical axis informationassociated with their imaging devices using motor controllers and/orposition encoders associated with the imaging devices, as well asimaging characteristics associated with the imaging devices.

Moreover, the control station may receive imaging data from theplurality of aerial vehicles, and/or any of the location, pose, and/oroptical axis information from the plurality of aerial vehicles. Inaddition, the control station may include an image processing module toperform image processing on the imaging data to identify objects anddetermine directional or optical rays along which the objects wereidentified. The image processing may include determining pixel locationsof objects within imaging data relative to optical axes associated withthe imaging data. Further, the image processing may include determiningdirectional or optical rays associated with the determined pixellocations of the objects based on imaging characteristics of the imagingdevices.

Further, the control station may include a three-dimensional mappingmodule to generate a three-dimensional map of the determined directionalor optical rays and identify any intersections, e.g., by applyingtriangulation processes or methods to the determined optical rays,associated with objects that were captured within imaging data of aplurality of aerial vehicles. Then, based on the identifiedintersections, the control station may determine locations and/or rangesof the identified objects relative to at least some of the plurality ofaerial vehicles.

FIG. 1 is a schematic diagram of an example object tracking system 100,according to an implementation.

The example object tracking system 100 may include a plurality of aerialvehicles 105, in which each of the plurality of aerial vehicles includesone or more imaging devices 107 to capture imaging data of one or moreobjects 110. In addition, each of the plurality of aerial vehicles 105may be in communication with an object tracking control system 120 via anetwork 115. Although FIG. 1 includes two aerial vehicles 105-1, 105-2having imaging devices 107-1, 107-2 to capture images of an object 110,any other number, combination, or arrangement of aerial vehicles 105,imaging devices 107, objects 110, and/or control systems 120 may formpart of the object tracking system 100.

As shown in FIG. 1, each aerial vehicle 105 may be any particular typeor configuration of aerial vehicle. For example, aerial vehicle 105-1 isshown as a quadcopter, and aerial vehicle 105-2 is shown as anoctocopter. In addition, the aerial vehicles 105 may be unmanned orautomated aerial vehicles (AAVs). Other types or configurations ofaerial vehicles may also be included in the object tracking system 100.

In addition, each aerial vehicle 105 may include one or more imagingdevices 107. For example, the imaging devices 107 may include analogcameras, digital cameras, video cameras, imaging sensors, infraredimaging devices, or other imaging devices. The imaging devices 107 maycapture imaging data in any frequency or wavelength of light, such ashuman-visible light, ultraviolet light, infrared light, or otherfrequencies or wavelengths.

Further, each aerial vehicle may determine and/or provide informationrelated to the aerial vehicle location, aerial vehicle pose ororientation, imaging device pose or orientation from which optical axesof one or more imaging devices may be determined, imaging data from oneor more imaging devices, and/or characteristics associated with the oneor more imaging devices. All or a portion of this information may beprovided to the object tracking control system 120 via the network 115.

The network 115 may be any communication network via which the pluralityof aerial vehicles 105 may communicate with the object tracking controlsystem 120. For example, the network 115 may include wireless or wirednetworks, and may include WiFi networks, LAN (local area networks), WAN(wide area networks), cellular communication networks, satellitecommunication networks, the Internet, or other networks.

The object tracking control system 120 may process the informationreceived from the plurality of aerial vehicles 105 to determinelocations and/or ranges of objects 110 relative to at least some of theplurality of aerial vehicles 105. For example, the object trackingcontrol system 120 may perform image processing on the received imagingdata to identify objects 110, determine pixel locations of the objects110 within the imaging data relative to optical axes associated with theimaging devices 107, and determine directional or optical rays that areassociated with the pixel locations of the objects 110 based at least inpart on characteristics of the imaging devices 107.

Alternatively, all or a portion of the image processing may be performedby individual aerial vehicles 105 prior to providing information to theobject tracking control system 120 via the network 115. In embodimentsin which all or a portion of the image processing is performed byindividual aerial vehicles 105, imaging data from the one or moreimaging devices and/or characteristics associated with the one or moreimaging devices may not need to be provided to the object trackingcontrol system 120. Further, in such embodiments, one or more outputs ofthe image processing, such as identified objects, determined pixellocations of the objects, and/or determined optical rays associated withthe pixel locations may be provided by the individual aerial vehicles105 to the object tracking control system 120.

Based at least in part on the determined optical rays associated withpixel locations of objects, and information related to aerial vehiclelocations, aerial vehicle poses or orientations, and/or imaging deviceposes or orientations from which optical axes of one or more imagingdevices may be determined, the object tracking control system 120 maygenerate a three-dimensional mapping of the determined optical rays anddetermine whether any two or more of the optical rays intersects with athreshold degree of confidence, e.g., by applying triangulationprocesses or methods to the determined optical rays. If two or moreoptical rays are determined to intersect, it may be determined thatobjects identified by two or more aerial vehicles associated with thetwo or more optical rays are actually the same object, and locationsand/or ranges of the object relative to the two or more aerial vehiclesmay be determined.

FIG. 2 is a schematic diagram 200 of an example aerial vehicle 205having an imaging device 207, according to an implementation.

The aerial vehicle 205 may include a frame or body, one or more imagingdevices 207 attached to the frame or body, a plurality of motors 208 andpropellers 209 also attached to the frame or body, e.g., via motor arms,a plurality of power supplies 212, and an aerial vehicle control system220. The frame or body may include any structure that supports thevarious components of the aerial vehicle 205, and may include a fuselageor central portion and a plurality of motor arms. Although FIG. 2 showsan aerial vehicle 205 having a quadcopter configuration, any other typeor configuration of aerial vehicle may be used in the systems andmethods described herein, e.g., a hexacopter, an octocopter, or otherconfiguration.

The one or more imaging devices 207 may include analog cameras, digitalcameras, video cameras, imaging sensors, infrared imaging devices, orother imaging devices. The imaging devices 207 may capture imaging datain any frequency or wavelength of light, such as human-visible light,ultraviolet light, infrared light, or other frequencies or wavelengths.In addition, the imaging devices 207 may be attached to the frame orbody, e.g., to the fuselage or central portion or to motor arms.Further, the imaging devices 207 may have a fixed pose or orientationrelative to the frame or body, or may have a variable or movable pose ororientation relative to the frame or body. For example, the pose ororientation may be defined by a direction of an optical axis 217 of animaging device 207, in which the optical axis 217 extends along acentral optical ray within the field of view of the imaging device 207,relative to the frame or body. Although FIG. 2 shows a single imagingdevice 207 attached to the fuselage of the aerial vehicle 205, any othernumber or arrangement of imaging devices 207 may be included in theaerial vehicles 205 as part of the systems and methods described herein.

The motors 208 and propellers 209 may be any motor and propellercombinations suitable to provide lift and/or maneuverability to theaerial vehicle 205. The motors 208 and propellers 209 may be attached tothe frame or body, e.g., to the fuselage or central portion or to motorarms. In addition, other propulsion mechanisms may be used in place ofor in combination with motors and propellers, such as fans, jets,turbojets, turbo fans, jet engines, electric jets, or others. AlthoughFIG. 2 shows four motors 208 and propellers 209, any other number orarrangement of propulsion mechanisms may be included in the aerialvehicles 205 as part of the systems and methods described herein.

The power supplies 212 may provide power to various components of theaerial vehicle 205, including the motors 208 and propellers 209, theimaging devices 207, the aerial vehicle control system 220, and anyother components. For example, the power supplies 212 may comprisebatteries having electrochemical cells of various types, such as lithiumion (Li-ion, LiPo, LIP, Li-poly or others), nickel metal hydride (NiMH),or other cell types. Although FIG. 2 shows two power supplies 212, anyother number or arrangement of power supplies 212 may be included in theaerial vehicles 205 as part of the systems and methods described herein.

The aerial vehicle control system 220, as further described herein withrespect to FIG. 8, may control the operation of the various componentsof the aerial vehicle 205, including navigation, operation of one ormore imaging devices 207, processing of imaging data captured by the oneor more imaging devices 207, communication with other aerial vehicles205, communication with a control station and/or an object trackingcontrol system 120, and other operations. Moreover, the aerial vehiclecontrol system 220 may receive instructions from the control stationand/or the object tracking control system 120 and control navigation oroperations of the aerial vehicle 205 based at least in part on thereceived instructions.

The aerial vehicle control system 220 may also include various sensorsto aid navigation or other operations. For example, the aerial vehiclecontrol system 220 may include location sensors, e.g., globalpositioning system (GPS) sensors, to determine a location of the aerialvehicle 205. Alternatively or in addition, other types of locationsensors may also be included in the aerial vehicle 205, such as localpositioning system sensors, altimeters, barometers, or other sensors.The location sensors may provide location information of the aerialvehicle 205 in terms of latitude, longitude, and altitude coordinatesrelative to the Earth, or other local coordinate systems.

In addition, the aerial vehicle control system 220 may include pose ororientation sensors to determine a pose or orientation of the aerialvehicle 205. For example, the pose or orientation sensors may includeinertial measurement units, accelerometers, gyroscopes, or othersensors. Based at least in part on information from the pose ororientation sensors, roll 222, pitch 224, and/or yaw 226 of the aerialvehicle 205, e.g., relative to the Earth, may be determined.

Although FIG. 2 shows particular numbers and arrangements of componentsincluded in the aerial vehicle 205, any other numbers and arrangementsof components may be included in the aerial vehicles 205 as part of thesystems and methods described herein. Further, the aerial vehicle 205may include other components not particularly shown or described, suchas various other sensors to aid navigation or other operations, and/oractuators or motors associated with imaging devices to change fields ofview and optical axes of the imaging devices.

FIG. 3 is a schematic diagram 300 of another example aerial vehicle 305having an imaging device 307, according to an implementation.

The aerial vehicle 305 may include the same or similar components asdescribed herein with respect to the aerial vehicle 205. For example,the aerial vehicle 305 may include a frame or body, one or more imagingdevices 307 attached to the frame or body, a plurality of motors 208 andpropellers 209 also attached to the frame or body, e.g., via motor arms,a plurality of power supplies 212, and an aerial vehicle control system220.

As shown in the FIG. 3, the imaging device 307 may be variably ormovably attached to the frame or body. For example, the imaging device307 may include one or more actuators or motors to change a field ofview and an optical axis of the imaging device 307. The imaging device307 may be attached to the frame or body of the aerial vehicle 305 viaone or more arms, beams, rods, plates, discs, joints, gimbals, or othermovable components such that the field of the view of the imaging device307 may be selectively modified. In addition, the movable components maybe moved via one or more actuators or motors, such as servos, solenoids,rotary actuators, linear actuators, pneumatic actuators, hydraulicactuators, or other actuators or motors.

Further, a position of the variable or movable imaging device 307relative to the frame or body of the aerial vehicle 305 may bedetermined using one or more sensors. For example, motor controllers,position encoders, rotary encoders, or other sensors may receive and/ordetermine a current position of the variable or movable imaging device307 relative to the frame or body of the aerial vehicle 305. Based atleast in part on the determined current position of the variable ormovable imaging device 307 relative to the frame or body of the aerialvehicle 305, an optical axis 317 of the current field of view of theimaging device 307 relative to the frame or body of the aerial vehicle305 may be determined. For example, the optical axis 317 may be anoptical ray that extends from a center point of the imaging device 307to a center point of the current field of view of the imaging device307.

Although FIG. 3 shows particular numbers and arrangements of componentsincluded in the aerial vehicle 305, any other numbers and arrangementsof components may be included in the aerial vehicles 305 as part of thesystems and methods described herein. Further, the aerial vehicle 305may include other components not particularly shown or described, suchas various other structures, motors, or actuators to change fields ofview and optical axes of imaging devices, and/or various other sensorsto determine current positions, fields of view, or optical axes ofimaging devices.

FIG. 4 is a schematic diagram 400 of an example field of view of animaging device 407, according to an implementation.

As shown in FIG. 4, an imaging device 407, whether fixedly or movablyattached to the frame or body of an aerial vehicle, may include acurrent field of view 428. The field of view 428 may be rectangular andmay have an aspect ratio of 4:3 with a respective width or horizontalextent that is larger than a respective height or vertical extent.Alternatively, the field of view 428 may have any other shape or aspectratio dependent upon imaging characteristics of the imaging device 407,e.g., shape, size, resolution, or other characteristics. For example,the field of view 428 may be square, circular, elliptical, or any othershape or size.

The imaging device 407 may also include an optical axis 417. Forexample, the optical axis 417 may be a central optical ray that extendsfrom a center point of the imaging device 407 to a center point 427 ofthe current field of view 428 of the imaging device 407. In someembodiments, the optical axis 417 may extend from a center point of alens of the imaging device 407 to the center point 427 of the currentfield of view 428. The optical axis 417 may be dependent upon imagingcharacteristics of one or more lenses of the imaging device 407, e.g.,shape, size, curvature, focal length, or other characteristics.

As shown in FIG. 4, an object 430 may also be present within the currentfield of view 428 of the imaging device 407. For example, the object 430may be a bird, other animal, an aerial vehicle, an airborne object, orany other object or structure. In addition, the object 430 may bepresent at one or more pixel locations within the current field of view428 relative to the optical axis. Based at least in part on the one ormore pixel locations of the object 430, a directional or optical ray 432associated with the one or more pixel locations may be determined. Thedirectional or optical ray 432 associated with the one or more pixellocations may be dependent upon imaging characteristics of the imagingdevice 407, e.g., shape, size, resolution, or other characteristics,and/or imaging characteristics of one or more lenses of the imagingdevice 407, e.g., shape, size, curvature, focal length, or othercharacteristics.

In example embodiments, the imaging device 407 and any associated lensesmay have a particular imaging resolution. Each pixel location of theimaging device 407 may be associated with a particular optical ray thatextends from the imaging device 407 to the pixel location. Given ahigher imaging resolution of the imaging device 407, a captured imagemay include a higher number of pixel locations with an associated highernumber of optical rays. Likewise, given a lower imaging resolution ofthe imaging device 407, a captured image may include a lower number ofpixel locations with an associated lower number of optical rays.

FIGS. 5A and 5B are schematic diagrams 500-1, 500-2 of example aerialvehicles as part of an object tracking system, according to animplementation.

As shown in FIG. 5A, an aerial vehicle 505-1 and an aerial vehicle 505-2may form part of an object tracking system, in combination with acontrol station or object tracking control system. The aerial vehicle505-1 may be at a particular location with a particular pose ororientation, and an imaging device of the aerial vehicle 505-1 may havea current field of view 528-1, as schematically indicated in FIG. 5A.The imaging device of the aerial vehicle 505-1 may also include a centerpoint 527-1 associated with an optical axis of the current field of view528-1 of the imaging device. An image captured by the imaging device ofthe aerial vehicle 505-1 may include an object 510-1, e.g., a bird orother object, that is located at a pixel location 530-1 relative to theoptical axis. In addition, based at least in part on the pixel location530-1, an optical ray 532-1 associated with the pixel location 530-1 maybe determined based on imaging characteristics of the imaging device andany associated lenses.

Further, as shown in FIG. 5A, the aerial vehicle 505-2 may be at aparticular location with a particular pose or orientation, and animaging device of the aerial vehicle 505-2 may have a current field ofview 528-2, as schematically indicated in FIG. 5A. The imaging device ofthe aerial vehicle 505-2 may also include a center point 527-2associated with an optical axis of the current field of view 528-2 ofthe imaging device. An image captured by the imaging device of theaerial vehicle 505-2 may also include the object 510-1, e.g., a bird orother object, that is located at a pixel location 530-2 relative to theoptical axis. In addition, based at least in part on the pixel location530-2, an optical ray 532-2 associated with the pixel location 530-2 maybe determined based on imaging characteristics of the imaging device andany associated lenses.

As further described herein, the optical rays 532-1, 532-2 may be mappedin three-dimensional space based least in part on locations of theaerial vehicles 505-1, 505-2, poses of the aerial vehicles 505-1, 505-2,directions of optical axes of imaging devices of the aerial vehicles505-1, 505-2, and/or pixel locations 530-1, 530-2 of objects identifiedin images captured by the imaging devices of the aerial vehicles 505-1,505-2. If it is determined that the optical rays 532-1, 532-2 intersectin three-dimensional space with a threshold degree of confidence basedat least in part on the three-dimensional mapping, then it may bedetermined that each of the aerial vehicles 505-1, 505-2 has captured animage of the same object 510-1. Then, a location and/or a range of theobject 510-1 relative to one or more of the aerial vehicles 505-1, 505-2may be determined based at least in part on the three-dimensionalmapping.

Accordingly, although each aerial vehicle individually may not be ableto determine a location and/or a range of an object relative to theaerial vehicle based on images captured by an imaging device associatedwith the individual aerial vehicle alone, the combination of imagingdata from two or more aerial vehicles that includes images of the sameobject, in combination with additional information related to the aerialvehicles and the imaging devices, may allow a determination of alocation and/or a range of the object relative to the two or more aerialvehicles.

As shown in FIG. 5B, an aerial vehicle 505-1, an aerial vehicle 505-2,and an aerial vehicle 505-3 may form part of an object tracking system,in combination with a control station or object tracking control system.The aerial vehicle 505-1 may be at a particular location with aparticular pose or orientation, and an imaging device of the aerialvehicle 505-1 may have a current field of view 528-1, as schematicallyindicated in FIG. 5B. The imaging device of the aerial vehicle 505-1 mayalso include a center point 527-1 associated with an optical axis of thecurrent field of view 528-1 of the imaging device. An image captured bythe imaging device of the aerial vehicle 505-1 may include an object510-1, e.g., a bird or other object, that is located at a pixel location530-1 relative to the optical axis. In addition, based at least in parton the pixel location 530-1, an optical ray 532-1 associated with thepixel location 530-1 may be determined based on imaging characteristicsof the imaging device and any associated lenses.

Further, as shown in FIG. 5B, the aerial vehicle 505-2 may be at aparticular location with a particular pose or orientation, and a firstimaging device of the aerial vehicle 505-2 may have a current field ofview 528-2, as schematically indicated in FIG. 5B. The first imagingdevice of the aerial vehicle 505-2 may also include a center point 527-2associated with an optical axis of the current field of view 528-2 ofthe first imaging device. An image captured by the first imaging deviceof the aerial vehicle 505-2 may also include the object 510-1, e.g., abird or other object, that is located at a pixel location 530-2 relativeto the optical axis. In addition, based at least in part on the pixellocation 530-2, an optical ray 532-2 associated with the pixel location530-2 may be determined based on imaging characteristics of the firstimaging device and any associated lenses.

Moreover, as shown in FIG. 5B, the aerial vehicle 505-3 may be at aparticular location with a particular pose or orientation, and animaging device of the aerial vehicle 505-3 may have a current field ofview 528-3, as schematically indicated in FIG. 5B. The imaging device ofthe aerial vehicle 505-3 may also include a center point 527-3associated with an optical axis of the current field of view 528-3 ofthe imaging device. An image captured by the imaging device of theaerial vehicle 505-3 may also include the object 510-1, e.g., a bird orother object, that is located at a pixel location 530-3 relative to theoptical axis. In addition, based at least in part on the pixel location530-3, an optical ray 532-3 associated with the pixel location 530-3 maybe determined based on imaging characteristics of the imaging device andany associated lenses.

As further described herein, the optical rays 532-1, 532-2, 532-3 may bemapped in three-dimensional space based least in part on locations ofthe aerial vehicles 505-1, 505-2, 505-3, poses of the aerial vehicles505-1, 505-2, 505-3, directions of optical axes of imaging devices ofthe aerial vehicles 505-1, 505-2, 505-3, and/or pixel locations 530-1,530-2, 530-3 of objects identified in images captured by the imagingdevices of the aerial vehicles 505-1, 505-2, 505-3. If it is determinedthat the optical rays 532-1, 532-2, 532-3 intersect in three-dimensionalspace with a threshold degree of confidence based at least in part onthe three-dimensional mapping, then it may be determined that each ofthe aerial vehicles 505-1, 505-2, 505-3 has captured an image of thesame object 510-1. Then, a location and/or a range of the object 510-1relative to one or more of the aerial vehicles 505-1, 505-2, 505-3 maybe determined based at least in part on the three-dimensional mapping.

In addition, as shown in FIG. 5B, the aerial vehicle 505-2 may be at aparticular location with a particular pose or orientation, and a secondimaging device of the aerial vehicle 505-2 may have a current field ofview 528-4, as schematically indicated in FIG. 5B. The second imagingdevice of the aerial vehicle 505-2 may also include a center point 527-4associated with an optical axis of the current field of view 528-4 ofthe second imaging device. An image captured by the second imagingdevice of the aerial vehicle 505-2 may include an object 510-2, e.g., atower or other object, that is located at a pixel location 530-4relative to the optical axis. In addition, based at least in part on thepixel location 530-4, an optical ray 532-4 associated with the pixellocation 530-4 may be determined based on imaging characteristics of thesecond imaging device and any associated lenses.

Further, as shown in FIG. 5B, the aerial vehicle 505-3 may be at aparticular location with a particular pose or orientation, and theimaging device of the aerial vehicle 505-3 may have a current field ofview 528-3, as schematically indicated in FIG. 5B. The imaging device ofthe aerial vehicle 505-3 may also include a center point 527-3associated with an optical axis of the current field of view 528-3 ofthe imaging device. An image captured by the imaging device of theaerial vehicle 505-3 may also include the object 510-2, e.g., the toweror other object, that is located at a pixel location 530-5 relative tothe optical axis. In addition, based at least in part on the pixellocation 530-5, an optical ray 532-5 associated with the pixel location530-5 may be determined based on imaging characteristics of the imagingdevice and any associated lenses.

As further described herein, the optical rays 532-4, 532-5 may be mappedin three-dimensional space based least in part on locations of theaerial vehicles 505-2, 505-3, poses of the aerial vehicles 505-2, 505-3,directions of optical axes of imaging devices of the aerial vehicles505-2, 505-3, and/or pixel locations 530-4, 530-5 of objects identifiedin images captured by the imaging devices of the aerial vehicles 505-2,505-3. If it is determined that the optical rays 532-4, 532-5 intersectin three-dimensional space with a threshold degree of confidence basedat least in part on the three-dimensional mapping, then it may bedetermined that each of the aerial vehicles 505-2, 505-3 has captured animage of the same object 510-2. Then, a location and/or a range of theobject 510-2 relative to one or more of the aerial vehicles 505-1,505-2, 505-3 may be determined based at least in part on thethree-dimensional mapping.

Further, as shown in FIG. 5B, an individual aerial vehicle, e.g., aerialvehicle 505-2, may include a plurality of imaging devices with differentor partially overlapping fields of view to capture images of objects anddetermine locations and/or ranges of the objects relative to one or moreaerial vehicles. In addition, an individual aerial vehicle, e.g., aerialvehicle 505-3, may capture representations of a plurality of objectswithin a single image and determine locations and/or ranges of theobjects relative to one or more aerial vehicles.

Accordingly, although each aerial vehicle individually may not be ableto determine a location and/or a range of an object relative to theaerial vehicle based on images captured by an imaging device associatedwith the individual aerial vehicle alone, the combination of imagingdata from two or more aerial vehicles that includes images of the sameobject, in combination with additional information related to the aerialvehicles and the imaging devices, may allow a determination of alocation and/or a range of the object relative to the two or more aerialvehicles.

Moreover, using the systems and methods described herein, locationsand/or ranges of particular objects relative to one or more aerialvehicles that have not captured any images of the particular objects mayalso be determined based at least in part on the three-dimensionalmapping. For example, as shown in FIG. 5B, based at least in part on thecaptured images from imaging devices of aerial vehicles 505-2, 505-3that include representations of the object 510-2, a location and/or arange of the object 510-2 relative to aerial vehicle 505-1 may also bedetermined, even in the absence of any imaging data captured by theimaging device of aerial vehicle 505-1 that includes the object 510-2,based at least in part on the three-dimensional mapping of the opticalrays 532-4, 532-5 and location and/or pose information associated withthe aerial vehicles 505-1, 505-2, 505-3 and their respective imagingdevices.

FIG. 6 is a flow diagram illustrating an example object directional raydetermination process 600, according to an implementation.

The process 600 may begin by receiving location information associatedwith an aerial vehicle, as at 602. For example, the location informationmay be received from and/or determined by various sensors of the aerialvehicle, such as GPS sensors, local positioning system sensors,barometers, altimeters, or other sensors. The location information mayinclude latitude, longitude, and/or altitude coordinates relative to theEarth, or other local coordinate systems.

The process 600 may then proceed by receiving pose informationassociated with the aerial vehicle, as at 604. For example, the poseinformation may be received from and/or determined by various sensors ofthe aerial vehicle, such as inertial measurement units, accelerometers,gyroscopes, or other sensors. The pose information may include roll,pitch, and/or yaw information of the aerial vehicle relative to theEarth.

The process 600 may then continue to receive optical axis information ofan imaging device on the aerial vehicle, as at 606. For example, a poseor orientation of the imaging device relative to a frame or body of theaerial vehicle may be received. If the imaging device is fixedlyattached to the frame or body of the aerial vehicle, then the pose ofthe imaging device may also be fixed, such that the optical axis of theimaging device is also fixed. Alternatively, if the imaging device ismovable relative to the frame or body of the aerial vehicle, the pose ofthe imaging device may be received from and/or determined by varioussensors of the aerial vehicle, such as motor controllers, positionencoders, rotary encoders, or other sensors. The optical axis of themovable imaging device may be determined based at least in part on thecurrent pose of the imaging device, in which the optical axis may be anoptical ray that extends from a center point of the imaging device to acenter point of the current field of view of the imaging device.

The process 600 may then proceed to receive imaging data from theimaging device on the aerial vehicle, as at 608. For example, theimaging data may include one or more still images or video, and may bereceived from and/or captured by various types of imaging devices, suchas analog cameras, digital cameras, video cameras, imaging sensors,infrared imaging devices, or other imaging devices.

The process 600 may then proceed to perform image processing on theimaging data to determine a pixel location of an object within theimaging data relative to the optical axis of the imaging device, as at610. For example, the object within the imaging data may be identifiedusing any image recognition methods or algorithms, such as edgerecognition, object recognition, or other similar algorithms. For thesystems and methods described herein, it may not be necessary toidentify any particular object, e.g., to identify an object as a bird ora tower. Instead, it may be sufficient to identify that an object ofsome unidentified type is present within the imaging data. Then, a pixellocation of the object within the imaging data relative to the opticalaxis may be identified, which may be dependent upon imagingcharacteristics of the imaging device, e.g., shape, size, resolution, orother characteristics.

The process 600 may then continue to determine a directional or opticalray associated with the pixel location of the object, as at 612. Forexample, dependent upon imaging characteristics of the imaging deviceand/or any associated lenses, e.g., shape, size, curvature, focallength, or other characteristics, a particular optical ray associatedwith the pixel location of the object may be determined.

Based at least in part on the location, pose, and/or optical axisinformation associated with the aerial vehicle, the particular opticalray associated with the pixel location of the object may be specificallyoriented in three-dimensional space. For example, by starting with thelocation of the aerial vehicle, then taking into account the pose of theaerial vehicle and the direction of the optical axis of the imagingdevice relative to the frame or body of the aerial vehicle, and thenfurther taking into account the pixel location of the object relative tothe optical axis of the imaging device, the particular optical rayassociated with the pixel location of the object may have a specificorientation in three-dimensional space with a starting point proximatethe location of the aerial vehicle and extending into three-dimensionalspace along the particular optical ray.

In some embodiments, an object may be associated with a plurality ofpixel locations clustered relatively close together, e.g., if theresolution of the imaging device is relatively high, if the object isrelatively close to the imaging device, and/or if the object isrelatively large in size. In such embodiments, a plurality of opticalrays associated with the plurality of pixel locations of the object maybe determined as corresponding to the object. The process 600 may thenend, as at 614.

Although FIG. 6 is generally described in terms of process steps thatmay be performed by a control station or object tracking control system120, all or portions of the process steps of FIG. 6 may be performed byone or more of the aerial vehicles. For example, if an aerial vehiclecontrol system of an aerial vehicle includes an image processing moduleto perform image processing on imaging data captured by an imagingdevice associated with the aerial vehicle, then all or portions ofprocess steps 610 and/or 612 may be performed by the aerial vehiclecontrol system using location, pose, and/or optical axis information ofthe aerial vehicle, and imaging data from the imaging device. In suchembodiments, none or only a portion of the location, pose, and/oroptical axis information, and/or imaging data may need to be provided toa control station for further processing and three-dimensional mappingof the determined optical rays. In other embodiments, the process stepsof FIG. 6 may be performed by one or more aerial vehicles and one ormore control stations in various other combinations.

FIG. 7 is a flow diagram illustrating an example object location andrange determination process 700, according to an implementation.

The process 700 may begin by performing the object directional raydetermination process, as described with respect to FIG. 6, for imagingdata from a plurality of imaging devices on a plurality of aerialvehicles, as at 702. For example, first imaging data may be receivedfrom a first imaging device associated with a first aerial vehicle, andsecond imaging data may also be received from a second imaging deviceassociated with a second aerial vehicle. The object directional raydetermination process 600 may be performed for each of the first imagingdata and second imaging data to determine optical rays associated withobjects identified in each set of imaging data.

The process 700 may continue by generating a three-dimensional mappingof the determined optical rays, as at 704. For example, thethree-dimensional mapping may be generated by a three-dimensionalmapping module of a control station or object tracking control system120 using the location, pose, and/or optical axis information from theaerial vehicles, as well as the determined optical rays associated withpixel locations of objects, which are determined at least in part by animage processing module. Then, the three-dimensional mapping module maycalculate, process, and/or build a three-dimensional rendering orrepresentation of a space including the determined optical rays. Theoptical rays may begin proximate locations of the aerial vehicles inthree-dimensional space and extend along the determined optical raysinto three-dimensional space.

The process 700 may then proceed to determine whether an intersectionbetween at least two of the determined optical rays has been identified,as at 706. For example, the three-dimensional mapping module of acontrol station or object tracking control system 120 may determinewhether at least two optical rays that have been calculated, processed,and/or built into a three-dimensional rendering or representation of aspace intersect with a threshold degree of confidence, e.g., by applyingtriangulation processes or methods to the optical rays.

The threshold degree of confidence may be determined based on variousfactors, such as absolute distance, relative distance, image resolution,other factors, or combinations thereof. For example, a threshold degreeof confidence based at least in part on an absolute distance maydetermine that two optical rays intersect if the minimum distancebetween the two optical rays is less than a defined absolute distance,e.g., 2 meters, 1 meter, 0.5 meter, or other absolute distance. In someembodiments, a threshold degree of confidence based at least in part ona relative distance may determine that two optical rays intersect basedon relative minimum distances between two optical rays versus otheroptical rays, e.g., two optical rays having a minimum distancetherebetween that is smaller than a minimum distance between any othertwo optical rays. In other embodiments, a threshold degree of confidencebased at least in part on a relative distance may determine that twooptical rays intersect based on a minimum distance between the twooptical rays at a potential point of intersection relative to distancesbetween starting points of the two optical rays and the potential pointof intersection. In still other embodiments, a threshold degree ofconfidence based at least in part on image resolution may determine thattwo optical rays intersect based on resolutions of imaging data withinwhich objects were identified at particular pixel locations, such thatimaging data having lower resolution may require two optical rays tohave a smaller minimum distance therebetween than two optical raysdetermined from imaging data having higher resolution. Other factorsthat may influence the threshold degree of confidence related todetermining intersections between at least two optical rays may relateto accuracy or precision of location information, pose information,and/or optical axis information, as well as other factors.

If it is determined at 706 that at least two optical rays do notintersect with a threshold degree of confidence, then the process 700may instruct aerial vehicles to navigate based on potential locations ofobjects along the determined optical rays, as at 708. For example,because no intersections between optical rays have been identified, eachidentified object may be present at any point along the identifiedoptical rays, such that the aerial vehicles may be instructed to avoidpotential conflicts with the identified optical rays in the absence offurther information or imaging data. The process 700 may then return to702 to repeat the process with any new or additional information orimaging data.

If, however, it is determined at 706 that at least two optical rays dointersect with a threshold degree of confidence, then the process 700may determine locations and/or ranges of objects associated with the atleast two optical rays relative to one or more of the aerial vehicles,as at 710. For example, the three-dimensional mapping module maydetermine locations and/or ranges of the objects based at least in parton the generated three-dimensional mapping, using the location, pose,and/or optical axis information from the aerial vehicles, as well as thedetermined optical rays.

Then, the process 700 may instruct aerial vehicles to navigate based onthe determined locations and/or ranges of objects associated with the atleast two optical rays relative to one or more of the aerial vehicles.For example, because an intersection between optical rays has beenidentified, each identified object may be present at a particularlocation in three-dimensional space and/or at a particular range from anaerial vehicle in three-dimensional space along the identified opticalrays, such that the aerial vehicles may be instructed to avoid potentialconflicts with the determined locations and/or ranges of the objects.The process 700 may then return to 702 to repeat the process with anynew or additional information or imaging data.

FIG. 8 is a block diagram illustrating various components of an aerialvehicle control system 220, according to an implementation. In variousexamples, the block diagram may be illustrative of one or more aspectsof the aerial vehicle control system 220 that may be used to implementthe various systems and processes discussed above. In the illustratedimplementation, the aerial vehicle control system 220 includes one ormore processors 802, coupled to a non-transitory computer readablestorage medium 820 via an input/output (I/O) interface 810. The aerialvehicle control system 220 may also include a propulsion controller 804,a power controller/supply module 806 and/or a navigation system 808. Theaerial vehicle control system 220 may further include an imaging devicecontroller 812, an image processing module 814, a network interface 816,and one or more input/output devices 818.

In various implementations, the aerial vehicle control system 220 may bea uniprocessor system including one processor 802, or a multiprocessorsystem including several processors 802 (e.g., two, four, eight, oranother suitable number). The processor(s) 802 may be any suitableprocessor capable of executing instructions. For example, in variousimplementations, the processor(s) 802 may be general-purpose or embeddedprocessors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, orany other suitable ISA. In multiprocessor systems, each processor(s) 802may commonly, but not necessarily, implement the same ISA.

The non-transitory computer readable storage medium 820 may beconfigured to store executable instructions, data, location information,pose information, imaging device information and characteristics,optical axis information, imaging data, pixel locations, directional oroptical rays, and/or other data items accessible by the processor(s)802. In various implementations, the non-transitory computer readablestorage medium 820 may be implemented using any suitable memorytechnology, such as static random access memory (SRAM), synchronousdynamic RAM (SDRAM), nonvolatile/Flash-type memory, or any other type ofmemory. In the illustrated implementation, program instructions and dataimplementing desired functions, such as those described above, are shownstored within the non-transitory computer readable storage medium 820 asprogram instructions 822, data storage 824 and other information anddata 826, respectively. In other implementations, program instructions,data and/or other information and data may be received, sent or storedupon different types of computer-accessible media, such asnon-transitory media, or on similar media separate from thenon-transitory computer readable storage medium 820 or the aerialvehicle control system 220.

Generally speaking, a non-transitory, computer readable storage mediummay include storage media or memory media such as magnetic or opticalmedia, e.g., disk or CD/DVD-ROM, coupled to the aerial vehicle controlsystem 220 via the I/O interface 810. Program instructions and datastored via a non-transitory computer readable medium may be transmittedby transmission media or signals, such as electrical, electromagnetic,or digital signals, which may be conveyed via a communication mediumsuch as a network and/or a wireless link, such as may be implemented viathe network interface 816.

In one implementation, the I/O interface 810 may be configured tocoordinate I/O traffic between the processor(s) 802, the non-transitorycomputer readable storage medium 820, and any peripheral devices, thenetwork interface 816 or other peripheral interfaces, such asinput/output devices 818. In some implementations, the I/O interface 810may perform any necessary protocol, timing or other data transformationsto convert data signals from one component (e.g., non-transitorycomputer readable storage medium 820) into a format suitable for use byanother component (e.g., processor(s) 802). In some implementations, theI/O interface 810 may include support for devices attached throughvarious types of peripheral buses, such as a variant of the PeripheralComponent Interconnect (PCI) bus standard or the Universal Serial Bus(USB) standard, for example. In some implementations, the function ofthe I/O interface 810 may be split into two or more separate components,such as a north bridge and a south bridge, for example. Also, in someimplementations, some or all of the functionality of the I/O interface810, such as an interface to the non-transitory computer readablestorage medium 820, may be incorporated directly into the processor(s)802.

The propulsion controller 804 may communicate with the navigation system808 and may adjust the operational characteristics of each propulsionmechanism to guide the aerial vehicle along a determined path and/or toperform other navigational maneuvers. The power controller/supply 806may include one or more power supplies, e.g., batteries, and may controlthe provision of power to various components of the aerial vehicle andthe aerial vehicle control system 220. The navigation system 808 mayinclude GPS sensors, other location sensors, inertial measurement units,accelerometers, gyroscopes, and/or other sensors than can be used tonavigate the aerial vehicle to and/or from a location.

The imaging device controller 812 may communicate with one or moreimaging devices included on the aerial vehicle, and may controloperation and/or movement of the imaging devices. The image processingmodule 814 may communicate with the imaging device controller 812 and/orthe one or more imaging devices, and may perform processing of theimaging data from the imaging devices, such as identifying objectswithin the imaging data, determining pixel locations of objects relativeto optical axes of the imaging devices, and/or determining optical raysassociated with the determined pixel locations.

The network interface 816 may be configured to allow data to beexchanged between the aerial vehicle control system 220, other devicesattached to a network, such as other computer systems, aerial vehiclecontrol systems of other aerial vehicles, control stations and/or objecttracking control systems 120. For example, the network interface 816 mayenable wireless communication between numerous aerial vehicles, controlstations, and/or object tracking control systems 120. In variousimplementations, the network interface 816 may support communication viawireless general data networks, such as a Wi-Fi network. For example,the network interface 816 may support communication viatelecommunications networks such as cellular communication networks,satellite networks, and the like.

Input/output devices 818 may, in some implementations, include one ormore visual output devices, audio input/output devices, input devicessuch as touchscreens, keyboards, or mice, image capture devices, thermalsensors, infrared sensors, time of flight sensors, location sensors,accelerometers, pressure sensors, weather sensors, other sensorsdescribed herein, etc. Multiple input/output devices 818 may be presentand controlled by the aerial vehicle control system 220.

As shown in FIG. 8, the memory may include program instructions 822which may be configured to implement the example processes and/orsub-processes described above. The data storage 824 and otherinformation and data 826 may include various data stores for maintainingdata items that may be provided for determining locations and/or posesof aerial vehicles, determining optical axes of imaging devices,processing imaging data, determining pixel locations of objects relativeto optical axes, determining directional or optical rays associated withpixel locations of objects, and any other functions, operations, orprocesses described herein.

FIG. 9 is a block diagram illustrating various components of an objecttracking control system 120, according to an implementation. In variousexamples, the block diagram may be illustrative of one or more aspectsof the object tracking control system 120 that may be used to implementthe various systems and processes discussed above. In the illustratedimplementation, the object tracking control system 120 includes one ormore processors 910, coupled to a non-transitory computer readablestorage medium 920 via an input/output (I/O) interface 930. The objecttracking control system 120 may also include an imaging processingmodule 932, a 3-D mapping module 934, a network interface 940, and oneor more input/output devices 950.

The object tracking control system 120 may be included as part of acontrol station, and the control station may form a part of the objecttracking system 100, in combination with a plurality of aerial vehicles.In some embodiments, the control station may be a fixed building,structure, or installation. In other embodiments, the control stationmay be mobile, and may comprise a ground-based vehicle, air-basedvehicle, a water-based vehicle, or other mobile structure orinstallation. In still other embodiments, one or more of the pluralityof aerial vehicles may include all or a portion of the componentsdescribed herein with respect to the object tracking control system 120,such that all or a portion of the operations of the object trackingcontrol system 120 may be performed by one or more aerial vehiclesand/or may be distributed among a plurality of aerial vehicles invarious combinations.

In various implementations, the object tracking control system 120 maybe a uniprocessor system including one processor 910, or amultiprocessor system including several processors 910A-910N (e.g., two,four, eight, or another suitable number). The processor(s) 910 may beany suitable processor capable of executing instructions. For example,in various implementations, the processor(s) 910 may be general-purposeor embedded processors implementing any of a variety of instruction setarchitectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, orany other suitable ISA. In multiprocessor systems, each processor(s) 910may commonly, but not necessarily, implement the same ISA.

The non-transitory computer readable storage medium 920 may beconfigured to store executable instructions, data, aerial vehiclelocation information, aerial vehicle pose information, imaging deviceinformation and characteristics, optical axis information, imaging data,pixel locations, directional or optical rays, object locations, objectranges, and/or other data items accessible by the processor(s) 910. Invarious implementations, the non-transitory computer readable storagemedium 920 may be implemented using any suitable memory technology, suchas static random access memory (SRAM), synchronous dynamic RAM (SDRAM),nonvolatile/Flash-type memory, or any other type of memory. In theillustrated implementation, program instructions and data implementingdesired functions, such as those described above, are shown storedwithin the non-transitory computer readable storage medium 920 asprogram instructions 922, data storage 924 and other information anddata 926, respectively. In other implementations, program instructions,data and/or other information and data may be received, sent or storedupon different types of computer-accessible media, such asnon-transitory media, or on similar media separate from thenon-transitory computer readable storage medium 920 or the objecttracking control system 120.

Generally speaking, a non-transitory, computer readable storage mediummay include storage media or memory media such as magnetic or opticalmedia, e.g., disk or CD/DVD-ROM, coupled to the object tracking controlsystem 120 via the I/O interface 930. Program instructions and datastored via a non-transitory computer readable medium may be transmittedby transmission media or signals, such as electrical, electromagnetic,or digital signals, which may be conveyed via a communication mediumsuch as a network and/or a wireless link, such as may be implemented viathe network interface 940.

In one implementation, the I/O interface 930 may be configured tocoordinate I/O traffic between the processor(s) 910, the non-transitorycomputer readable storage medium 920, and any peripheral devices, thenetwork interface 940 or other peripheral interfaces, such asinput/output devices 950. In some implementations, the I/O interface 930may perform any necessary protocol, timing or other data transformationsto convert data signals from one component (e.g., non-transitorycomputer readable storage medium 920) into a format suitable for use byanother component (e.g., processor(s) 910). In some implementations, theI/O interface 930 may include support for devices attached throughvarious types of peripheral buses, such as a variant of the PeripheralComponent Interconnect (PCI) bus standard or the Universal Serial Bus(USB) standard, for example. In some implementations, the function ofthe I/O interface 930 may be split into two or more separate components,such as a north bridge and a south bridge, for example. Also, in someimplementations, some or all of the functionality of the I/O interface930, such as an interface to the non-transitory computer readablestorage medium 920, may be incorporated directly into the processor(s)910.

The image processing module 932 may perform processing of the imagingdata from the imaging devices of aerial vehicles, such as identifyingobjects within the imaging data, determining pixel locations of objectsrelative to optical axes of the imaging devices, and/or determiningoptical rays associated with the determined pixel locations. Thethree-dimensional mapping module 934 may perform processing to calculateand/or build three-dimensional representations or renderings of spacesusing aerial vehicle location information, aerial vehicle poseinformation, and/or optical axis information, as well as pixel locationsand/or optical rays associated with identified objects, to identifyintersections between optical rays with a threshold degree ofconfidence, and/or to determine locations and/or ranges of objectsrelative to one or more aerial vehicles.

The network interface 940 may be configured to allow data to beexchanged between the object tracking control system 120, other devicesattached to a network, such as other computer systems, aerial vehiclecontrol systems of aerial vehicles, other control stations and/or otherobject tracking control systems 120. For example, the network interface940 may enable wireless communication between numerous aerial vehicles,control stations, and/or object tracking control systems 120. In variousimplementations, the network interface 940 may support communication viawireless general data networks, such as a Wi-Fi network. For example,the network interface 940 may support communication viatelecommunications networks such as cellular communication networks,satellite networks, and the like.

Input/output devices 950 may, in some implementations, include one ormore visual output devices, audio input/output devices, input devicessuch as touchscreens, keyboards, or mice, image capture devices, othersensor described herein, etc. Multiple input/output devices 950 may bepresent and controlled by the object tracking control system 120.

As shown in FIG. 9, the memory may include program instructions 922which may be configured to implement the example processes and/orsub-processes described above. The data storage 924 and otherinformation and data 926 may include various data stores for maintainingdata items that may be provided for determining locations and/or posesof aerial vehicles, determining optical axes of imaging devices,processing imaging data, determining pixel locations of objects relativeto optical axes, determining directional or optical rays associated withpixel locations of objects, generating three-dimensional mappinginformation, identifying intersections between optical rays, determininglocations and/or ranges of identified objects, instructing aerialvehicles, and any other functions, operations, or processes describedherein.

In various implementations, the parameter values and other dataillustrated herein as being included in one or more data stores may becombined with other information not described or may be partitioneddifferently into more, fewer, or different data structures. In someimplementations, data stores may be physically located in one memory ormay be distributed among two or more memories.

Each process described herein may be implemented by the architecturesdescribed herein or by other architectures. The processes areillustrated as a collection of blocks in a logical flow. Some of theblocks represent operations that can be implemented in hardware,software, or a combination thereof. In the context of software, theblocks represent computer-executable instructions stored on one or morecomputer readable media that, when executed by one or more processors,perform the recited operations. Generally, computer-executableinstructions include routines, programs, objects, components, datastructures, and the like that perform particular functions or implementparticular abstract data types.

The computer readable media may include non-transitory computer readablestorage media, which may include hard drives, floppy diskettes, opticaldisks, CD-ROMs, DVDs, read-only memories (ROMs), random access memories(RAMs), EPROMs, EEPROMs, flash memory, magnetic or optical cards,solid-state memory devices, or other types of storage media suitable forstoring electronic instructions. In addition, in some implementations,the computer readable media may include a transitory computer readablesignal (in compressed or uncompressed form). Examples of computerreadable signals, whether modulated using a carrier or not, include, butare not limited to, signals that a computer system hosting or running acomputer program can be configured to access, including signalsdownloaded through the Internet or other networks. Finally, the order inwhich the operations are described is not intended to be construed as alimitation, and any number of the described operations can be combinedin any order and/or in parallel to implement the process. Additionally,one or more of the operations may be considered optional and/or notutilized with other operations.

Those skilled in the art will appreciate that the aerial vehicle controlsystem 220 and the object tracking control system 120 are merelyillustrative and are not intended to limit the scope of the presentdisclosure. In particular, the computing systems and devices may includeany combination of hardware or software that can perform the indicatedfunctions, including computers, network devices, internet appliances,PDAs, wireless phones, pagers, etc. The aerial vehicle control system220 and the object tracking control system 120 may also be connected toother devices that are not illustrated, or instead may operate asstand-alone systems. In addition, the functionality provided by theillustrated components may, in some implementations, be combined infewer components or distributed in additional components. Similarly, insome implementations, the functionality of some of the illustratedcomponents may not be provided and/or other additional functionality maybe available.

Those skilled in the art will also appreciate that, while various itemsare illustrated as being stored in memory or storage while being used,these items or portions of them may be transferred between memory andother storage devices for purposes of memory management and dataintegrity. Alternatively, in other implementations, some or all of thesoftware components may execute in memory on another device andcommunicate with the illustrated aerial vehicle control system 220 orthe object tracking control system 120. Some or all of the systemcomponents or data structures may also be stored (e.g., as instructionsor structured data) on a non-transitory, computer-accessible medium or aportable article to be read by an appropriate drive, various examples ofwhich are described above. In some implementations, instructions storedon a computer-accessible medium separate from the aerial vehicle controlsystem 220 or the object tracking control system 120 may be transmittedto the aerial vehicle control system 220 or the object tracking controlsystem 120 via transmission media or signals, such as electrical,electromagnetic, or digital signals, conveyed via a communicationmedium, such as a network and/or a wireless link. Variousimplementations may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Accordingly, thetechniques described herein may be practiced with other aerial vehiclecontrol system or other object tracking control system configurations.

Those skilled in the art will appreciate that, in some implementations,the functionality provided by the processes and systems discussed abovemay be provided in alternative ways, such as being split among moresoftware modules or routines or consolidated into fewer modules orroutines. Similarly, in some implementations, illustrated processes andsystems may provide more or less functionality than is described, suchas when other illustrated processes instead lack or include suchfunctionality respectively, or when the amount of functionality that isprovided is altered. In addition, while various operations may beillustrated as being performed in a particular manner (e.g., in serialor in parallel) and/or in a particular order, those skilled in the artwill appreciate that, in other implementations, the operations may beperformed in other orders and in other manners. Those skilled in the artwill also appreciate that the data structures discussed above may bestructured in different manners, such as by having a single datastructure split into multiple data structures or by having multiple datastructures consolidated into a single data structure. Similarly, in someimplementations, illustrated data structures may store more or lessinformation than is described, such as when other illustrated datastructures instead lack or include such information respectively, orwhen the amount or types of information that is stored is altered. Thevarious processes and systems as illustrated in the figures anddescribed herein represent example implementations. The processes andsystems may be implemented in software, hardware, or a combinationthereof in other implementations. Similarly, the order of any processmay be changed, and various elements may be added, reordered, combined,omitted, modified, etc., in other implementations.

From the foregoing, it will be appreciated that, although specificimplementations have been described herein for purposes of illustration,various modifications may be made without deviating from the spirit andscope of the appended claims and the features recited therein. Inaddition, while certain aspects are presented below in certain claimforms, the inventors contemplate the various aspects in any availableclaim form. For example, while only some aspects may currently berecited as being embodied in a computer readable storage medium, otheraspects may likewise be so embodied. Various modifications and changesmay be made as would be obvious to a person skilled in the art havingthe benefit of this disclosure. It is intended to embrace all suchmodifications and changes and, accordingly, the above description is tobe regarded in an illustrative rather than a restrictive sense.

What is claimed is:
 1. A computer-implemented method to track airborneobjects, comprising: receiving, from a first aerial vehicle of aplurality of aerial vehicles: first location information associated withthe first aerial vehicle; first pose information associated with thefirst aerial vehicle; first optical axis information associated with afirst imaging device of the first aerial vehicle; and first imaging datafrom the first imaging device of the first aerial vehicle; identifying afirst object within the first imaging data; determining a first pixellocation of the first object within the first imaging data; determininga first optical ray associated with the first pixel location of thefirst object; receiving, from a second aerial vehicle of the pluralityof aerial vehicles: second location information associated with thesecond aerial vehicle; second pose information associated with thesecond aerial vehicle; second optical axis information associated with asecond imaging device of the second aerial vehicle; and second imagingdata from the second imaging device of the second aerial vehicle;identifying a second object within the second imaging data; determininga second pixel location of the second object within the second imagingdata; determining a second optical ray associated with the second pixellocation of the second object; determining that the first optical rayand the second optical ray intersect with a threshold degree ofconfidence, such that the first object and the second object are a sameobject; determining at least one of a location or a range of the sameobject relative to at least one of the first aerial vehicle or thesecond aerial vehicle; and instructing the at least one of the firstaerial vehicle or the second aerial vehicle to navigate to avoid thesame object based at least in part on the determined at least one of thelocation or the range of the same object relative to the at least one ofthe first aerial vehicle or the second aerial vehicle.
 2. Thecomputer-implemented method of claim 1, wherein the first locationinformation includes first latitude, longitude, and altitudeinformation, and the second location information includes secondlatitude, longitude, and altitude information.
 3. Thecomputer-implemented method of claim 1, wherein the first poseinformation includes first roll, pitch, and yaw information, and thesecond pose information includes second, roll, pitch, and yawinformation.
 4. The computer-implemented method of claim 1, wherein thefirst optical axis information includes a first direction of a firstoptical axis of the first imaging device relative to a first frame ofthe first aerial vehicle, and the second optical axis informationincludes a second direction of a second optical axis of the secondimaging device relative to a second frame of the second aerial vehicle.5. A computer-implemented method to track objects, comprising:receiving, from a plurality of aerial vehicles, imaging data fromrespective imaging devices of individual aerial vehicles; identifyingobjects within the imaging data; determining pixel locations of theobjects within the imaging data; determining optical rays associatedwith the pixel locations of the objects; determining that at least twooptical rays intersect with a threshold degree of confidence based on atleast one of location, pose, or optical axis information associated withindividual aerial vehicles; determining that at least two objectsassociated with the at least two intersecting optical rays are a sameobject; determining at least one of a location or a range of the sameobject relative to at least one of the plurality of aerial vehicles; andinstructing the at least one of the plurality of aerial vehicles tonavigate to avoid the same object based at least in part on thedetermined at least one of the location or the range of the same objectrelative to the at least one of the plurality of aerial vehicles.
 6. Thecomputer-implemented method of claim 5, further comprising: receiving,from the plurality of aerial vehicles, the at least one of location,pose, or optical axis information associated with individual aerialvehicles.
 7. The computer-implemented method of claim 6, wherein thelocation information includes latitude, longitude, and altitudeinformation received from at least one of a global positioning system(GPS) sensor or an altimeter associated with individual aerial vehicles.8. The computer-implemented method of claim 6, wherein the poseinformation includes roll, pitch, and yaw information received from atleast one of an inertial measurement unit, an accelerometer, or agyroscope associated with individual aerial vehicles.
 9. Thecomputer-implemented method of claim 6, wherein the optical axisinformation includes directions of fixed optical axes associated withrespective imaging devices of individual aerial vehicles.
 10. Thecomputer-implemented method of claim 6, wherein the optical axisinformation includes directions of movable optical axes received from atleast one of a motor controller or a position encoder associated withrespective imaging devices of individual aerial vehicles.
 11. Thecomputer-implemented method of claim 5, wherein the determining pixellocations of the objects within the imaging data further comprises:determining a pixel location associated with an object identified withinthe imaging data received from a respective imaging device relative toan optical axis associated with the respective imaging device.
 12. Thecomputer-implemented method of claim 5, wherein the determining opticalrays associated with the pixel locations of the objects furthercomprises: determining an optical ray associated with a pixel locationof an object identified within the imaging data received from arespective imaging device based on at least one characteristic of therespective imaging device.
 13. The computer-implemented method of claim5, wherein the determining that at least two optical rays intersect witha threshold degree of confidence based on at least one of location,pose, or optical axis information associated with individual aerialvehicles further comprises: generating a three-dimensional mapping ofthe optical rays based on the at least one of location, pose, or opticalaxis information associated with individual aerial vehicles; anddetermining that the at least two optical rays intersect with thethreshold degree of confidence based on the three-dimensional mapping.14. The computer-implemented method of claim 13, wherein the determiningat least one of a location or a range of the same object relative to atleast one of the plurality of aerial vehicles further comprises:determining the at least one of the location or the range of the sameobject relative to the at least one of the plurality of aerial vehiclesbased on the three-dimensional mapping.
 15. A system to track objects,comprising: a plurality of aerial vehicles having respective imagingdevices; and a control system in communication with the plurality ofaerial vehicles and having a processor configured to at least: receive,from the plurality of aerial vehicles, imaging data from respectiveimaging devices of individual aerial vehicles; identify objects withinthe imaging data; determine pixel locations of the objects within theimaging data; determine optical rays associated with the pixel locationsof the objects; determine that at least two optical rays intersect witha threshold degree of confidence based on at least one of location,pose, or optical axis information associated with individual aerialvehicles; determine that at least two objects associated with the atleast two intersecting optical rays are a same object; determine atleast one of a location or a range of the same object relative to atleast one of the plurality of aerial vehicles; and instruct the at leastone of the plurality of aerial vehicles to navigate to avoid the sameobject based at least in part on the determined at least one of thelocation or the range of the same object relative to the at least one ofthe plurality of aerial vehicles.
 16. The system of claim 15, whereinthe control system is at least one of a ground-based control system, anair-based control system, or a water-based control system.
 17. Thesystem of claim 15, wherein the control system is at least one of fixedor mobile.
 18. The system of claim 15, wherein the control system iscomprised in at least one of the plurality of aerial vehicles.
 19. Thesystem of claim 15, wherein the control system includes at least one ofan image processing module or a three-dimensional mapping module.